1. Technical Field
This disclosure relates to estimating spectral distributions of reflections from points on an object surface, including specular roughness and tangent vectors.
2. Description of Related Art
Measuring the appearance of real materials may be described by a bidirectional reflectance distribution function (BRDF). See Nicodemus F. E., Richmond J. C., Hsia J. J., Ginsberg I. W., Limperis T, Geometric considerations and nomenclature for reflectance, National Bureau of Standards Monograph 160 (1977). The function may be a 4D function that relates the ratio of reflectance between the incident and outgoing directions for a single surface point.
These BRDF models may depend on a sparse set of non-linear parameters that roughly correspond to albedo, specular roughness, surface normal, and tangent directions. Measuring and fitting these parameters for a particular material model may require a dense sampling of incident and outgoing lighting directions. Fitting the parameters of an a-priori chosen BRDF model to observed measurements may rely on complex fragile non-linear optimization procedures.
Some methods may estimate fundamental parameters of appearance, such as normal direction and albedo, without assuming an a-priori material model. Instead they may rely on general properties shared by many physical materials such as symmetry. See Zickler T. E., Belhumeur P. N., Kriegman D. J., Helmholtz stereopsis: Exploiting reciprocity for surface reconstruction, Int. J. Comput. Vision 49, 2-3 (2002), 215-227; Alldrin N., Kriegman D., Toward reconstructing surfaces with arbitrary isotropic reflectance: A stratified photometric stereo approach, Proceedings of the International Conference on Computer Vision (ICCV) (2007), pp. 1-8; Ma W.-C., Hawkins T., Peers P., Chabert C.-F., Weiss M., Debevec P., Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination, Rendering Techniques (2007), pp. 183-194; Alldrin N., Zickler T., Kriegman D., Photometric stereo with non-parametric and spatially-varying reflectance, Proceedings of IEEE Computer Vision and Pattern Recognition (CVPR) (2008); Holroyd M., Lawrence J., Humphreys G., Zickler T., A photometric approach for estimating normals and tangents, SIGGRAPH Asia '08: ACM SIGGRAPH Asia 2008 papers (2008), pp. 1-9.
The first order spherical statistics of the reflectance under distant illumination may correspond to the normal and reflection vector for diffuse and specular materials respectively. See In Ma W.-C., Hawkins T., Peers P., Chabert C.-F., Weiss M., Debevec P., Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination, Rendering Techniques (2007), pp. 183-194. The first order statistics may be efficiently measured using linear gradient illumination conditions. See Ma et al., supra.
The representation and acquisition of the appearance of physical materials and objects may often be classified in two categories: (1) Analytical BRDF-based methods; and (2) Non-parametric and photometric methods.
Analytical BRDF-Based Methods
Analytical material representation BRDF may be designed with certain goals in mind, such as physical accuracy, see Torrance K. E., Sparrow E. M., Theory of off specular reflection from roughened surfaces, J. Opt. Soc. Am. 57 (1967), 1104-1114; He X. D., Torrance K. E., Sillion F. X., Greenberg D. P.: A comprehensive physical model for light reflection. SIGGRAPH Comput. Graph. 25, 4 (1991), 175-186; facilitating rendering, see Ashikhmin M., Premoze S., Shirley P. S.: A microfacet-based BRDF generator. Proceedings of ACM SIGGRAPH 2000 (2000), Computer Graphics Proceedings, Annual Conference Series, pp. 65-74; and versatility and flexibility in approximating physical materials, see Ashikhmin M, Premoze S., Distribution-based BRDFs, Technical Report 2007; Lafortune E. P. F., Foo S.-C., Torrance K. E., Greenberg D. P.: Non-linear approximation of reflectance functions. SIGGRAPH '97: Proceedings of the 24th annual conference on Computer graphics and interactive techniques (1997), pp. 117-126; Ward G. J.: Measuring and modeling anisotropic reflection. SIGGRAPH Comput. Graph. 26, 2 (1992), 265-272.
Spatially varying appearance; based on the above analytical BRDF models, may be captured to recreate complete digital copies of existing real world objects. See, e.g., Gardner A., Tchou C., Hawkins T., Debevec P.: Linear light source reflectometry. ACM SIGGRAPH 2003 (2003), pp. 749-758; Georghiades A.: Recovering 3-D shape and reflectance from a small number of photographs. Rendering Techniques (2003), pp. 230Ü-240; Goldman D. B., Curless B., Hertzmann A., Seitz S. M.: Shape and spatially-varying BRDFs from photometric stereo. ICCV '05: Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 (2005), pp. 341-348; Lensch H. P. A., Goesele M., Kautz J., Heidrich W., Seidel H.-P.: Image-based reconstruction of spatially varying materials. Proceedings of the 12th Eurographics Workshop on Rendering Techniques (2001), pp. 103-114; Lensch H. P. A., Kautz J., Goesele M., Heidrich W., Seidel H.-P.: Image-based reconstruction of spatial appearance and geometric detail. ACM Transactions on Graphics 22, 2 (2003), 234-257; Marschner S.: Inverse Rendering for Computer Graphics. PhD thesis, Cornell University, 1998.
Each of these methods may require either a special acquisition device or an acquisition scheme tailored towards a particular prefixed analytical material model. Each may sacrifice spatial variation for angular variation to reduce the total acquisition time. However, the choice of BRDF model may impact the quality of appearance reproduction significantly. See Ngan A., Durand F., Matusik W.: Experimental analysis of BRDF models. Proceedings of the Eurographics Symposium on Rendering (2005), pp. 117-226. Furthermore, fitting the model-specific parameters may be complicated and ill-conditioned due to the non-linear nature of the parameters and the presence of measurement noise. It may only become clear after attempting to fit the measured data that the choice of material model may be suboptimal.
Switching to a better suited BRDF model after the fact may be difficult due to the inherent reliance of the acquisition setup/scheme of these methods on a particular model.
Non-Parametric and Photometric Techniques
Classical photometric stereo, see Woodham R. J.: Photometric stereo: A reflectance map technique for determining surface orientation from image intensity. Proc. SPIE's 22nd Annual Technical Symposium (1978), vola 155, may estimate surface normals by assuming an underlying Lambertian material, and by using a small set of fixed viewpoint observations under point lighting. However, most materials may not be purely Lambertian, and thus an inaccurate surface normal may be estimated. As a result, photometric stereo has been extended to non-Lambertian materials. See, e.g., Georghiades A.: Recovering 3-D shape and reflectance from a small number of photographs. Rendering Techniques (2003), pp. 230Ü-240; Goldman D. B., Curless B., Hertzmann A., Seitz S. M.: Shape and spatially-varying brdfs from photometric stereo. ICCV '05: Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 (2005), pp. 341-348. While these methods may handle a wider range of material types, they still may rely on (mostly isotropic) analytical BRDF models that limit their generality.
To overcome this limitation, a number of techniques have been proposed that may avoid using parametric BRDF models. See Mallick S. P., Zickler T. E., Kriegman D. J., Belhumeur P. N.: Beyond lambert: Reconstructing specular surfaces using color. Proc. IEEE Conf. Computer Vision and Pattern Recognition (2005). Mallick et al. reduce a general material to Lambertian by removing the specular “component,” and subsequently apply traditional photometric stereo. Hertzmann and Seitz estimate surface normals using a reference object with known shape and similar material properties as the target object. See Hertzmann A., Seitz S.: Shape and materials by example: a photometric stereo approach. Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on 1 (2003), 533-540, vol. 1. While this method may not rely on parametric appearance models, it may require a reference object which is not always available.
Another class of methods may exploit general properties of surface reflectance to infer surface statistics. A common assumption is that the maximum reflectance is observed when the halfway vector coincides with the surface normal. See, e.g., Francken Y., Cuypers T., Mertens T., Gielis J., Bekaert P.: High quality mesostructure acquisition using specularities. Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on (2008), 1-7). Various forms of symmetry may constitute another property that has been exploited. Zickler et al. exploit the Helmholtz reciprocity to recover depth and normal directions. See Zickler T. E., Belhumeur P. N., Kriegman D. J.: Helmholtz stereopsis: Exploiting reciprocity for surface reconstruction. Int. J. Comput. Vision 49, 2-3 (2002), 215-227. Alldrin and Kriegman exploit the symmetry about the view-normal plane in case of isotropic BRDFs. See Alldrin N., Kriegman D.: Toward reconstructing surfaces with arbitrary isotropic reflectance: A stratified photometric stereo approach. Proceedings of the International Conference on Computer Vision (ICCV) (2007), pp. 1-8; Ma et al. assume symmetry about the mean vector of a BRDF observed from a fixed viewpoint. See Ma W.-C., Hawkins T., Peers P., Chabert C.-F., Weiss M., Debevec P.: Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. Rendering Techniques (2007), pp. 183-194. They show that for Lambertian reflections the mean may correspond to the surface normal, and for specular reflections to the reflected direction. Both may be efficiently and directly measured by using polarized spherical gradient illumination. Holroyd et al. assume a similar type of symmetry, but use a dense sampling to resolve both the normal direction as well as tangent vectors per surface point. See Holroyd M., Lawrence J., Humphreys G., Zickler T.: A photometric approach for estimating normals and tangents. In SIGGRAPH Asia '08: ACM SIGGRAPH Asia 2008 papers (2008), pp. 1-9.
While these methods may not rely on a parametric model, they may not provide complete information regarding the surface reflectance. Lawrence et al. used inverse shade trees and an optimization scheme coined ACLS to decompose the spatially varying material properties of planar samples from dense hemispherical samplings in a collection of 1D curves and 2D textures. See Lawrence J., Ben-Artzi A., Decoro C., Matusik W., Pfister H., Ramamoorthi R., Rusinkiewicz S.: Inverse shade trees for non-parametric material representation and editing. ACM Transactions on Graphics 25, 3 (2006), 735-745. Alldrin et al. also employ ACLS to compute bivariate representations of isotropic surface appearances, including surface normals. See Alldrin N., Zickler T., Kriegman D.: Photometric stereo with non-parametric and spatially-varying reflectance. Proceedings of IEEE Computer Vision and Pattern Recognition (CVPR) (2008). Zickler et al. share reflectance information from different surface points to create a dense nonparametric reconstruction of appearance. See Zickler T., Ramamoorthi R., Enrique S., Belhumeur P. N.: Reflectance sharing: Predicting appearance from a sparse set of images of a known shape. IEEE Trans. Pattern Anal. Mach. Intell. 28, 8 (2006), 1287-1302.
Image-Based Relighting
Image-based relighting is a technique for synthesizing images of a scene under novel illumination conditions, based on a set of input photographs. In its most basic form, images of a scene may be acquired, see Paul Haeberli, Synthetic lighting for photography, January 1992, or rendered, see Jeffry S. Nimeroff, Eero Simoncelli, and Julie Dorsey, Efficient re-rendering of naturally illuminated environments, Fifth Eurographics Workshop on Rendering, June 1994, pp. 359-373. under a set of basis lighting conditions. Then, a relit version of the scene may be produced by taking linear combinations of the basis lighting conditions, akin to compositing together different lighting passes of a model miniature.
Debevec et al. used a light stage device to acquire a dataset of a human face lit by a dense set of over two thousand lighting directions on the sphere, and showed that such datasets could be efficiently illuminated under novel real-world lighting conditions such as high dynamic range lighting environments through image-based relighting. See Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar, Acquiring the reflectance field of a human face, Proc. ACM SIGGRAPH 2000, Computer. Graphics Proceedings, Annual Conference Series, July 2000, pp. 145-156. Recent work in the area of pre-computed radiance transfer has shown that pre-rendering an object's reflectance under basis illumination conditions may allow for real-time relighting as it moves through interactive environments. See Peter-Pike Sloan, Jan Kautz, and John Snyder, Precomputed radiance transfer for real-time rendering in dynamic, lowfrequency lighting environments, ACM Transactions on Graphics 21 (2002), no. 3, 527-536; Ravi Ramamoorthi and Pat Hanrahan, Frequency space environment map rendering, ACM Transactions on Graphics 21 (2002), no. 3, 517-526; Rui Wang, John Tran, and David Luebke, All-frequency interactive relighting of translucent objects with single and multiple scattering, ACM Transactions on Graphics 24 (2005), no. 3, 1202-1207. Basis illumination datasets have also been shown to be useful for object recognition, see R. Ramamoorthi, Modeling Illumination Variation with Spherical Harmonics, Face Processing: Advanced Modeling Methods, 2006, pp. 385-424, including for faces.
A benefit to image-based relighting techniques may be that complex illumination effects including spatially-varying diffuse and specular reflection, self-shadowing, mutual illumination, and subsurface scattering may be all encoded within the data and thus may appear accurately in the renderings. Traditional techniques may require far more advanced reflectometry and light transport simulation. Drawbacks may include that a lot of data must be acquired and stored. This may make the techniques less practical for capturing dynamic subjects. For example, high-speed video at thousands of frames per second may be required for dynamic subjects, as in Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec, Performance relighting and reflectance transformation with timemultiplexed illumination, ACM Transactions on Graphics 24 (2005), no. 3, 756-764), and for memory-efficient relighting as hundreds of images may be required.
Efficient representations of Image-Based Relighting datasets has been explored for rendering, generally by focusing on efficient representations for the scene's per-pixel reflectance functions.
Debevec et al. estimated diffuse and specular albedos and normals for each pixel's reflectance function, reducing the information for each pixel from hundreds of reflectance measurements to just a few reflectance parameters. See Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar, Acquiring the reflectance field of a human face, Proc. ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, July 2000, pp. 145-156. However, these parameters attempted to factor out global illumination effects, which may require these effects to be simulated later and may forfeit the elegance and realism of image based relighting.
Malzbender et al. fit quadratic polynomial texture maps (PTMs) to reflectance functions consisting of fifty lighting directions across the hemisphere. See Tom Malzbender, Dan Gelb, and Hans Wolters, Polynomial texture maps, Proc. SIGGRAPH 2001, pp. 519-528. The PTMs may greatly reduce the reflectance data to a compact, data-driven representation, and resulting renderings may produce realistic and relatively smooth and diffuse renditions of the objects under varying illumination. However, the technique may still require a dense set of incident lighting directions to be recorded. Its consideration may also have been restricted to lighting originating from the front hemisphere, which may be a significant limitation for fully three-dimensional objects.
Ma et al. used a computational illumination approach to modeling reflectance functions using a small number of incident lighting conditions. See Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec, Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination, Rendering Techniques 2007: 18th Eurographics Symposium on Rendering, June 2007, pp. 183-194. Using a spherical light stage and a set of four spherical gradient illumination conditions derived from the 0th and 1st-order spherical harmonics, the technique may directly measure the magnitude (albedo) and centroid (surface normal) of each pixel's reflectance function. These measurements may be used to drive a Lambertian or Phong reflectance lobe to represent the reflectance function; using polarization difference imaging, the diffuse and specular components may be modeled independently. As with PTMs, the resulting renderings may still encode most of the effects of global illumination. However, the lobe widths may need to be selected manually (either choosing a Lambertian lobe or a Phong specular exponent), as no reflectance lobe width information may be derived from the measurements.
All of the above methods may either require a dense sampling of the lighting directions, integrate information over multiple surface points, or deliver incomplete appearance information.